Will our mortgage bank, real state agency or Title Services company involved in the purchase of the property back us up?
Check your Title policy. Often times there is a clause that they will not represent you re: adverse possession.
What are our rights here? What are the chances of him getting his way in court?
Speaking generally and without specificity to your State's laws, he has the potential for an easement/adverse possession claim against your property. I just finished this section two weeks ago in my property class. From now on, I will not let people do sh*t on my land. If you have been allowing him access through your land for 20 years, you have some serious legal issues now.
Do we need to lawyer up?
Yes, yes, yes. And you need to start asserting trespass everytime you find him on your land.
Here are some cases from my property case book:
Trespass/right to exclude:
Jacque v. Steenberg Homes, Inc. *read this one; most all of these are online; look for the "legal brief."
Adams v. Cleveland-Cliffs Iron Co.
Watchtower Bible and Tract Society of New York v. Village of Stratton *yes, we did review this in class related to property rights.
Adverse Possession
Tran v. Macha
Howard v. Kunto
Mannillo v. Gorski
Easements, Non--Express Creation
Thompson v. E.I.G
Schwab v. timmons
Marrone v. Washington Jockey Club